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PRODUCT OF RANDOM STOCHASTIC MATRICES* 

By Behrouz Touri and Angelia Nedic 
University of Illinois at Urbana- Champaign 



It is well-known that for any aperiodic and irreducible stochastic 
matrix A, limfe_>oo A'' exists and it is a rank one stochastic matrix. 
We show that a generalization of this result not only holds for in- 
homogeneous chains of stochastic matrices but also for independent 
random chains of such matrices. 



1. Introduction. Let (Jl, J^, Pr) be a probability space and let be a chain of m x m 

random row stochastic matrices, i.e. for all A; > 1, the matrix W{k) is a row stochastic almost surely 
and Wij{k) : — > M is a Borel- measurable function for all i,j € [m], where [m] = {!,..., m}. 
Throughout this paper, we denote random chains by last alphabet letters such as and 
{U{k)}, and we use the first alphabet letters such as {A{k)} and {B{k)} to denote deterministic 
stochastic chains. We will exclusively deal with row stochastic chains, so we will refer to them as 
stochastic chains, or just simply as chains. 
' Let {14^(A;)} be an independent random chain. Then, we say that {W^(A;)} is strongly aperiodic if 

' there exists a 7 € (0, 1] such that 

E[Wii{k)Wij{k)] > -fE[Wij{k)] for ah i / j G [m] and all k > 1. 

Note that if Wii{k) > 7 almost surely for all i € [m] and all /c > 1, then such a chain is strongly 
aperiodic. Also, note that by summing both sides of the above inequality over j ^ i, we obtain 



E[Wiiik)] > E[Wii{k){l - Wiiik))] > 7(1 - E[Wii{k)]). 



Hence, E[Wii(A;)] > for all i € [m] and all k > 1. Thus, for a strongly aperiodic chain 
the expected chain {E[VF(/c)]} is strongly aperiodic. It follows that a deterministic chain {A{k)} is 
strongly aperiodic if and only if Aii{k) > 7 for some 7 > 0, and for all i G [m] and k > 1. 

For the subsequent use, for an m x m random (or deterministic) matrix W and a non-trivial 
^ , index set S C [m] (i.e. S ^$ and S 7^ [m]), we define the quantity 1^55 = X]ie5ie5^«i' '^here S 

^ \ is the complement of the index set S. 

We say that an independent random chain {VF(A;)} is balanced if there exists some a > such 
that 

(1) E[Wss{k)\ > aE[Wss{k)] for all nontrivial 5 C [m] and all k > 1. 

From this definition, it can be seen that a < 1. 

Finally, with a given random chain {1/^(A;)}, let us associate a random graph G°° = {[m],£°°) 
with the vertex set [m] and the edge set given by 

£^{ij) = f^{ij} \f2{Wi,{k,u) +w,^{k,u)) = 
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We refer to G°° as the infinite flow graph of {TV (A;)}. By the Kolmogorov's 0-1 law, the infinite flow 
graph of an independent random chain is almost surely equal to a deterministic graph. 

It has been shown that this determinstic graph is equal to the infinite flow graph of the expected 
chain {E[T4^(A:)]} ([8], Theorem 5). 

For a matrix W , let Wi and denote the iih. row vector and the jth column vector of TV, 
respectively. Also, for a chain {W{k)}, let W{k : to) = W{k) ■ ■ ■ W{to + 1), where k > to > and 
let W{k : k) = I for all fc > 0. With these preliminary definitions and notation in place, we can 
state the main result of the current study. 

Theorem 1. Let {W{k)} he an independent random stochastic chain which is balanced and 
strongly aperiodic. Then, for any to ^ the product W{k : to) = W{k) ■ ■ ■ W{to + 1) converges to a 
random stochastic matrix W{oo : to), almost surely. Furthermore, for all i,j in the same connected 
component of the infinite flow graph of {W{k)}, we have Wi{oo : to) = Wj{oo : to) almost surely. 

As an immediate consequence of this result, it follows that W{oo : to) has rank at most r where 
T is the number of the connected components of the infinite flow graph G°° of {VF(fc)}. Thus, if 
G°° is a connected graph, the limiting random matrix W{oo : to) = lim^._>.oo W{k : to) is a rank-one 
random stochastic matrix almost surely, i.e. W{oo : to) = ev'^{to) almost surely for some stochastic 
vector v{to). This and Theorem 1 imply that: if an independent random chain {VF(A:)} is balanced 
and strongly aperiodic, then {^/^(A;)} is almost surely strongly ergodic (as defined in [3]) if and only 
if the infinite flow graph of {ly(fc)} is connected. 

2. The Dynamics System Perspective. In order to prove Theorem 1, we establish some 
intermediate results, some of which are applicable to a more general category of random stochastic 
chains, namely adapted random chains. For this, let be a random chain adapted to a 

filtration {J-k}- For an integer to > and a vector v € M™, consider the trivial random vector 
x{to) : — )• M™ defined by x{to,uj) = v for all w G ^2. Now, recursively define: 

(2) x{k + l) = W{k + l)x{k) for all /c> to. 

Note that x{to) is measurable with respect to the trivial cr-algebra {0, 0} and hence, it is measurable 
with respect to Tt^. Also, since {VF(fc)} is adapted to {J-k}i it follows that for any k > to, x{k) is 
measurable with respect to J-k- We refer to {x{k)} as a random dynamics driven by {H^(A:)} started 
at the initial point {to,v) G Z+ x R™. We say that a given property holds for any dynamics {x{k)} 
driven by if that property holds for any initial point (to,u) € x M™. 

Note that if limk-^ooW{k : to) = W{oo : to) exists almost surely, then for any initial point 
(to,'^') S X M™, the random dynamics {x{k)} converges to W{oo : to)v almost surely. Also, note 
that for any i,j G [m], we have limfc^oo \\Wi{k,to) — Wj{k,to)\\ = almost surely if and only if 
limfc_^oo {xi{k) — Xj{k)) = almost surely for any initial point. When verifying the latter relation, 
due to linearity of the dynamics, it suffices to check that limfc_>oo {xi{k) — Xj{k)) = for all the 
initial points of the form (to, ei) with i S [m], where {ei, . . . , Cm} is the standard basis for M"^. 

In order to study the limiting behavior of the products W{k : to), we study the limiting behavior 
of the dynamics {x{k)} driven by {VF(A;)}. This enables us to use the dynamics system's tools 
and its stability theory to draw conclusions about the limiting behavior of the matrix products 
W{k : to). 

2.1. Why the Infinite Flow Graph. In this section we provide a result showing the relevance 
of the infinite flow graph to the study of the product of stochastic matrices. Let us consider a 
deterministic chain {A{k)} of stochastic matrices and let us define mutual ergodicity and an ergodic 
index as follows. 
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Definition 1. For an mxm chain {A{k)} of stochastic matrices, we say that an index i € [m] 
is ergodic i/limfc_j.oo Ai{k : to) exists for all to ^ 0. Also, we say that two disctinct indices i,j € [m] 
are mutually ergodic i/lim/^.^oo : ^o) ~ Aj{k : to)|| = 0. 

From the definition it immediately follows that an index i E [m] is ergodic for a chain {A{k)} 
if and only if limfc_j.oo 2;i(A:) exists for any dynamics {x{k)} driven by {A{k)}. Similarly, indices 
i,j € [m] are mutually ergodic if and only if liui/^^oo {xi{k) — Xj{k)) = for any dynamics driven 
by {A{k)}. 

The following result illustrates the relevance of the infinite flow graph to the study of the products 
of stochastic matrices. 

Lemma 1. ([8], Lemma 2) Two distinct indices i,j € [m] are mutually ergodic only if i and j 
belong to the same connected component of the infinite flow graph of{A{k)}. 

Generally, if i and j are mutually ergodic indices, it is not necessarily true that they are ergodic 
indices. As an example, consider the 4x4 stochastic chain {A(k)} defined by: 



A{2k) 



10 

10 

10 

1 



A{2k + 1) 



10 

1 

1 

1 



for all k > 1. 



It can be verified that for any starting time to >0 and any k > to, we have A{k : to) = A{k). Thus, 
it follows that indices 2 and 3 are mutually ergodic, while limfc_j.oo ^2(^ : ^o) and limfc^oo^3(^ : ^o) 
do not exist. 

The following result shows that under special circumstances, we can assert that some indices are 
ergodic if we know that a certain mutual ergodicity pattern exists in a chain. 

Lemma 2. Let S be a connected component of the infinite flow graph G°° of a chain {A{k)}. 
Suppose that indices i and j are mutually ergodic for all distinct i,j € S. Then, every index i S 
is ergodic. 

Proof. Without loss of generality let us assume that S = {1, . . . ,i*} for some i* € [m]. Let S 
be the complement of S. For the given chain {A(k)} and the connected component S, let the chain 
{B{k)} be defined by: 



B^Jik) 



Aij{k) if i ^ j and i,j £ S or i, j € S, 

if i j and z € S, j € <S or i G 5, j G 5, 

Aii{k) + J2eGS ^ieik) if i = j £ S_, 

[ Aii{k) + X:^e5 ^idk) iii = j£S. 



Then, B{k) has the block diagonal structure of the following form 



Bik) 



Bi{k) 
B2{k) 



for all k > 1. 



By construction the chain {B{k)} is stochastic. It can be verified that Yl'h=i ~ < 

for all i, j € [m]. Thus, {B{k)} is an ^i-approximation of {A{k)} as defined in [8]. Then, by Lemma 1 
in [8], it follows that indices i and j are mutually ergodic for the chain {B(k)} for all distinct i,j G S. 
By the block diagonal form of {B{k)}, it follows that i and j are mutually ergodic for the \S\ x \S\ 
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chain {Bi{k)} and all i,j G S. This, however, implies that the chain {Bi{k)} is weakly ergodic 
(as defined in [3]) and, as proven in Theorem 1 in [3], this further implies that {Bi{k)} is strongly 
ergodic, i.e. any index i £ S is ergodic for {Bi{k)}. Again, by the application of Lemma 1 in [8], 
we conclude that any index i £ S is ergodic for {A{k)}. Q.E.D. 

2.2. Comparison Functions. Here, we show that under general conditions, a rich family of com- 
parison functions exists for the dynamics {x{k)} driven by a random chain {1/F(A;)}. 

Let us define an absolute probability process for an adapted chain {VF(A;)}, which is an extension 
of the concept of the absolute probability sequence introduced by A. Kolmogorov for deterministic 
chains in [4]. 

Definition 2. We say that a random (vector) process {7r(A;)} is an absolute probability process 
for a random chain {W{k)} adapted to {Tk} if 

1. the random process {7r(A:)} is adapted to {Fk\, 

2. the vector 7r(fc) is stochastic almost surely for all k > 1, and 

3. the following relation holds almost surely 

E[7r^(A; + l)W{k + 1) | Tk] = vr^(/c) for all k>0. 

When an absolute probability process exists for a chain, we say that the chain admits an absolute 
probability process. 

For a deterministic chain of stochastic matrices {A{k)}, Kolmogorov showed in [4] that there 
exists a sequence of stochastic vectors {v{k)} such that v'^{k + l)A{k + 1) = v'^{k) for all A; > 0. 
Note that, for an independent random chain, any absolute probability sequence for the expected 
chain is an absolute probability process for the random chain. Thus, the existence of an absolute 
probability process for an independent random chain of stochastic matrices follows immediately 
from the Kolmogorov's existence result. As another non-trivial example of random chains that 
admit an absolute probability process, one may consider an adapted random chain that is 

doubly stochastic almost surely. In this case, the static sequence {^e} is an absolute probability 
process for {VF(A;)}, where e € M*" is the vector with all components equal to 1. 

Now, suppose that we have an adapted chain {VF(fc)} which admits an absolute probability 
sequence {7r(A;)}. Also, let : M — )■ M be an arbitrary convex function. Let us define the function 
Vg,^ : M'" X Z+ ^ E, as follows: 

m 

(3) Vg,^(x, k) = Y^ ■Ki{k)g{xi) - g{TT^{k)x) for all x G M"^ and ah k>0. 

i=l 

From the definition of an absolute probability process, it follows that Vg^Tr{x{k),k) is measurable 
with respect to J-^ for any dynamics {x{k)} driven by a chain {M^(A;)} that is adapted to {J-fc}. 
Also, since 7r(fc) is almost surely stochastic vector and g is a convex function, it follows that for any 
X G M™', we have V^,7r(x, k) > almost surely for all A: > 0. 

Next, we show that Vg^j^ is a comparison function for the dynamics (2) for any convex function 
g. In particular, we prove that {V^^^(x(A;), /c)} is a super-martingale sequence irrespective of the 
initial point for the dynamics {x{k)}. 

Theorem 2. Let {W{k)} be an adapted chain that admits an absolute probability process 
{'7T{k)}. Then, for the dynamics (2) started at any initial point {to,v) G x M™, we have 



E[Vg,^{x{k + 1), + 1) I Tk] < Vg,^ix{k), k) for all k > to. 
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Proof. By the definition of V^^tt in (3), we have almost surely 

m 

Vg^^{x{k + l),k + l) = Y, Mk + l)9{xi{k + 1)) - g{7T^{k + l)x{k + 1)) 

i=l 
m 

= TTiik + l)g{[W{k + l)x{k)]i) - <7(vr^(A; + l)x{k + 1)) 
1=1 

m m 

(4) < + 1) Wij{k + l)g{x,{k)) - g{T,^{k + l)x{k + 1)), 

i=i j=i 

where in the second equality we use to denote the ith component of a vector, while the in- 
equality is obtained by using the convexity of g{-) and the fact that matrix W{k) is stochas- 
tic almost surely. Since {7r(A;)} is an absolute probability process for {1^(A;)}, it follows that 
E[7r'^(A: + l)W{k + 1) | Fk\ = ir'^ik). Also, since x{k) is measurable with respect to T^, by tak- 
ing the conditional expectation with respect to J-k on both sides of Eq. (4), we obtain almost 
surely 

m 

E[Vg^n{xik + 1), A: + 1) I Tk] < Y^j{k)g{xj{k)) - E[<7(vr^(A: + l)x{k + 1)) | Tk\ 

m 

< Y 7T,{k)gixj{k)) - g{E[7r^{k + l)x{k + 1) | J^^] ), 
i=i 

where the last inequality follows by the convexity of g and Jensen's inequality. The result follows 
by using x{k + 1) = W{k + l)x{k) and the definition of absolute probability process. Q.E.D. 

Theorem 2 shows that the dynamics (2) admits infinitely many comparison functions, provided 
that {Ty(A;)} admits an absolute probability process. 

Since Vg^Tr{x{k),k) > almost surely for all /c > 0, it follows that {Vg^Tr{x{k), k)} is a bounded 
super-martingale. Hence, it is convergent almost surely irrespective of the initial point of the dy- 
namics {x{k)} and the choice of the convex function g. 

Corollary 1. Let {W{k)} be an adapted chain that admits an absolute probability process 
{7r(A;)}. Then, for any dynamics {x(A;)} driven by {W{k)} and for any convex function g : M ^ M, 
the limit limf^^^Vg^Tr{x{k),k) exists almost surely. 

2.3. Quadratic Comparison Function. In the sequel, we focus on the particular choice of function 
g{,s) = in relation (3). For convenience, we let 

■m m 

V^ix,k) = ^^,(fc)(x, - TT^{k)xf = YMk)x^ - {n^{k)xf. 

1=1 i=l 

For this particular choice of the convex function g, we can provide a lower bound for the decrease 
of the conditional expectations E[Vg{x{k + 1), /c + 1) | T^], which is exact under certain conditions. 

Theorem 3. Let {W{k)} be an adapted random chain with an absolute probability process 
{ir{k)}. Then, for any dynamics {x{k)} driven by {W{k)}, we have almost surely 

E[V^{x{k + 1), A; + 1) I Tk] < V^{x{k), k) H,j{k){xi{k) - Xj{k)f for all k > to, 

i<j 
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where H{k) = E[M^-^(A; + l)diag{7r{k + l))W{k + 1) | J-k] with diag{v) denoting the diagonal matrix 
induced by a vector v (i.e., with components vi on the main diagonal), and 'Yli<j ~ X^^ii THjLi+i- 
Furthermore, if tt^ [k + l)W{k + 1) = t:'^ [k) almost surely, then the inequality holds as an equality. 

Proof. We have for all k >to, 

m 

(5) V^{x{k),k) = TTi{k)xi{k) - {tt'^ {k)x{k)f = x^ {k)di&g{TT{k))x{k) - {tt^ {k)x{k)f . 

1=1 

Thus, by letting A{x{k), k) = V.„{x{k), k) - V-^{x{k + l),k + 1) and using x{k + 1) = W{k + l)x{k), 
we obtain for all k > t^, 

A{x{k),k) = x'^ {k)diag{-K{k))x{k) - {t:'^ {k)x{k)f 

- {x^{k + l)diag(7r(A: + l))x{k + 1) - {'k'^ {k + l)x{k + l))^} 
= x'^ik) [diag(7r(A;)) - W'^ {k + l)diag(7r(A; + l))W{k + 1)] x{k) 

+ {{^^{k + l)x{k + l)f - {T:^{k)x{k)f] 
= x'^{k)L{k)x{k) + {{7T^{k + l)x{k + l)f - {tt^ {k)x{k)f} , 

where L{k) = diag(7r(A:)) - W^{k + l)diag(7r(A; + l))W{k + 1). 

Note that the sequence {n'^ {k)x{k)} is a martingale, implying that {— (7r-^(/c)x(A:))^} is a super- 
martingale. Thus, by taking the conditional expectation on both sides of the preceding equality 
and noticing that x{k) is measurable with respect to J-^, we have almost surely 

(6) E[A{x{k),k) I Tk] > E[x'^{k)L{k)x{k) \ Tk] = x'^{k)E[L{k) \ Tk]x{k) for ah k > to. 

Further, letting e G be the vector with all components equal to 1, from the definition of L{k) 
we almost surely have for all k > to: 

E[L{k) \Tk]e = E[diag(^(A:))e - W^{k + l)diag(^(A: + l))W{k + l)e | J^k] 
= 7r{k) - E[W^{k + l)7r{k + 1) I Tk] = 0, 

which holds since W{k) is stochastic almost surely and {Tr{k)} is an absolute probability process for 
{M/^(/c)}. Thus, the random matrix E[L{k) \ J-^] is symmetric and E[L{k) \ J-^] e = almost surely. 
It can be shown that for a symmetric matrix A with Ae = 0, we have x'^Ax = — Aij[xi — Xj)'^ . 
Then, it follows that almost surely 

x^{k)E[L{k) I Tk]x{k) = -Y,Hij{k){xi{k)-Xj{k))\ 

i<j 

where H{k) = E[W'^{k + l)diag(7r(A; + l))W{k + 1) | /fc] . Using this relation in inequality (6), we 
conclude that almost surely 

(7) E[V^ix{k + 1), A: + 1) j Tk] < V^{x{k), A:) - ^ Hij{k){xi{k) - Xj{k)f for ah k > to. 

In the proof of inequality (7), the inequality appears due to relation (6) only. If 7r'^{k + l)W{k + 
1) = 7r'^{k)W{k) almost surely, then we have 7r^(fc + l)x{k + 1) = tt'^ (k)x{k) almost surely. Thus, 
relation (6) holds as an equality, and consequently so does relation (7). Q.E.D. 
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One of the important implications of Theorem 3 is the following result. 

Corollary 2. Let {W{k)} be an adapted random chain that admits an absolute probability 
process {7r(A;)}. Then, for any random dynamics {x{k)} driven by {W{k)}, we have for all to > 0, 



k=to i<j 



where L{k) = W^{k + l)diag{^{k + l))W{k + 1). 



< E[K(2;(to),to)] < oo, 



Proof. By taking expectation on both sides of the relation in Theorem 3, we obtain for all 
k > to: 



(8) 



E[V^{xik + l),k + 1)] < E[V^{x{k),k)] - E 



Since x{k) is measurable with respect to J^^, it follows that 

E[L,,{k) I J-fc] {x,{k) - Xj{k)f = E[U,{k){x,{k) - Xj{k)f I J-fc] = E[U,{k){x^{k) - Xj{k)f] . 
Using this relation in Eq. (8), we see that for all k >to: 



E[K(x(A; + 1), A: + 1)] < E[V^ix{k), k)] 



Y,Uj{k){xi{k) - Xj{k)f 
i<j 



Hence, ET=to^[T.i<j Uj{k){xi{k) - Xj{k)f\ < E[V^{x{to),to)] for any to > 0. Q.E.D. 

3. Class T'*. In this section, we introduce a class of random chains which we refer to it as the 
class V* and prove one of the central results of this work. In particular, we show that the claim of 
Theorem 1 holds for any chain that is in the class V* and satisfies some form of aperiodicity. 

Definition 3. The class V* is the class of random adapted chains that admit an absolute 
probability process {^{k)} that is unformly bounded away from zero almost surely, i.e., TTi{k) > p* 
almost surely for some scalar p* > 0, and for all k > and all i € [rn\. We write this concisely as 
{7r{k)} >p* >0. 

It may appear that the definition of the class V* is rather a restrictive requirement. Later on, 
we show that in fact the class V* contains a broad family of deterministic and random chains. 
To establish the main result of this section, we make use of the following intermediate result. 

Lemma 3. Let {A{k)} be a deterministic chain with the infinite flow graph G°° = {[m],£°°). 
Let {to,v) € Z+ X be an initial point for the dynamics driven by {A(k)}. If 

lim {xif^{k) - Xjo{k)) / 0, 

k—^oo 

for some io,jo belonging to the same connected component of , then we have 



Y}{A,j{k + 1) + Aji{k + mxi{k) - xj{k)f] = oo. 

k=tQ i<j 
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Proof. Let io,jo be in the same connected component of G°° with hm sup^-^oo (xj^ (k) — xj^ (k)) = 
a > 0. Without loss of generahty we may assume that x(to) £ [—1; 1]™) for otherwise we can con- 
sider the dynamics started at y(io) = ||^.(^^-| | | — x{tQ). Let S be the vertex set of the connected 
component in G°° containing io,jo, and without loss of generality assume that 5 = {1, 2, . . . , g} for 
some q G [m], q > 2. Then, by the definition of the infinite flow graph, there exists a large enough 
K > to such that 

oo 

k=K ^ 

where As{k + 1) = Ag§{k + 1) + A§g{k + 1). Furthermore, since limsupfc_j,oo — Xjg{k)) = 

a > 0, there exists a time instance ti > K such that x^^{ti) — XjQ{ti) > ^. 

Let (7 : [q] ^ [q] be a permutation such that Xf^(^i-^{ti) > x^^2){'ti) > • • • > ^^^.(^^(ii), i.e. a is an 
ordering of {xi{ti) \ i G [q]}. Since Xig{ti) — Xj^{ti) > ^, it follows that x^(^i-j{ti) — x^(^g-^{ti) > ^ 
and, therefore, there exists i G [q] such that x^(^£^{ti) — Xo-(£4-i)(ti) > Let 

t 

Ti = argminj; E (^.(^VO) + 1) + ^.0>W + 1)) > ^• 

^ k=ti i,je[g] 

i<e,i+i<j 

Since 5 is a connected component of the infinite flow graph G°°, we must have Ti < oo; otherwise, 
S could be decomposed into two disconnected components . . . ,cr{l)} and {a{l + l), . . . ,a{q)}. 

Now, let R = {cr(l), . . . , a{l)}. We have for any k G [ti, Ti]: 



Ti-l Ti-l ^ 



Ajiik + 1) = 



k=ti k=ti 



\ «>iG[g] 
\i leqe,£+l<j 



+ A^(^^j{k + 1)+ J2 Aaij){k + l) 



Ti-1 oo 

^ E E (^<x{.Mi)(fe + 1) + ^.aM.)(/c + 1)) + 5] As{k + 1) < 
i<e,e+i<j 

which follows by the definition of Ti and the choice of ti > K. By Lemma 1 in [9], it follows that 
for k G [ti,Ti], 

ca a 

max a;,- (A;) < maxx,(ti) + 2 , min Xjik) > min Xj(U) — 2 . 

ieR ^ i&R ^ ' 16g i&S\R ^ ^ - ieS\R ^ ^ 16q 



Thus, for any i,j G [q] with i < I and j > I + 1, and for any k G [ti, Ti], we have 

a \ a 
'iGq) ~ Jq' 



x.«(fc)-x.(,)(/c)>2(2-^') 
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Therefore, 



Yl Yl (^<T(iMi)(^ + l) + ^a(jMi)(^ + l))(a;<T(i)(^)-a^<T(i)(^))' 



i<i,£+l<j 



Further, it follows that: 

Ti 

YiMk + 1) + + mxi{k) - xj{k)f 

k=ti i<j 

Ti 

A:=ti i,j&[q] 
i<i,i+l<j 

Since lim sup^j^o^ (^jo (^) ~ ^joi^)) = a > 0, there exists a time t2 > Ti such that Xjo(t2) — 
Xjo{t2) > f • Then, using the above argument, there exists T2 > t2 such that Yl'kLt2 X]j<j(^ij(^ + 
1) + Aji{k + l))(xi(A;) — Xj{k))'^ > j3. Hence, using the induction, we can find time instances 

• • • > Tg+i > t^+i >T^>t^> T^^i > t^^i > ■■■ >Ti>ti>to, 

such that Y^li^ Ei<i(^u(^ + 1) + ^jiik + - 2;j(A;))2 > /3 for any ^ > 1- The intervals 

[ig,T^] are non-overlapping subintervals of [io,oo), implying that 

00 

J2 Y^Mk + 1) + M'' + - x,{k)f = 00. 

k=to i<j 

Q.E.D. 

For our main result, let us define the weak aperiodicity for an adapted random chain. 

Definition 4. We say that an adapted random chain {W{k)} is weakly aperiodic if for some 
7 > 0, and for all distinct i,j S [m] and all k >0, 

E[w'^{k + l)W'ik + 1) I Tk] > -/E[Wi,ik + 1) + Wji{k + 1) I Tk] . 

Now, we establish the main result of this section. 

Theorem 4. Let {W{k)} € V* be an adapted chain that is weakly aperiodic. Then, liuii^^oo W{k : 
to) = W[oo : to) exists almost surely for any t^ > 0. Moreover, the event under which Wi{oo : to) = 
Wj{oo : to) for all tQ > is almost surely equal to the event that i,j are belonging to the same 
connected component of the infinite flow graph of {W{k)}. 

Proof. Since is in V*, {W{k)} admits an absolute probability process {7r{k)} such that 

{7r(A;)} > p* > almost surely. Thus, it follows that 

p*E[W^{k + l)W{k + 1) I Jfc] < E[W^{k + l)diag(7r(A; + l))W{k + 1) | Tk] = H{k + 1). 
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On the other hand, by the weak aperiodicity, we have 

^E[W^J{k + 1) + Wji{k + 1) I J-fc] < E[W'^{k + l)W^{k + 1) I J-fc 



forsome7G (0,1] and for ah distinct G [m]. Thus, wehave p*^E[Wij{k + 1) + Wji{k + 1) \ J^k] < 
Hij{k + 1). By CoroUary 2, for the random dynamics {x{k)} driven by {l^(/c)} and started at ar- 
bitrary {to,v) e Z+ X W\ it fohows that 



CXD 



k=to 



Y,iWij{k) + Wj^ik)){xiik) - Xj{k)f 



< E[V^ix{to),to)]. 



As a consequence, 



''^{Wij{k) + Wji{k)){xi{k) — Xj{k))'^ < oo ahnost surely. 

k=to i<j 

Therefore, by Lemma 3, W6 conclude that liin^^^Qc {xi{k, oj) — Xj{k, uj)) = for any i, j belonging to 
the same connected component of G°^{oj), for almost all a; € fi. By Lemma 2 it follows that every 
index i G [m] is ergodic for almost all oj G Q. By considering the initial conditions (to, £ x I^™ 
for all £ G [m], the assertion follows. Q.E.D. 

Theorem 4 shows that the dynamics in (2) is convergent almost surely for aperiodic chains 
{Ty(fc)} G V* . Moreover, the theorem also characterizes the limiting points of such a dynamics as 
well as the limit matrices of the products W{k : to) as k ^ oo. 

4. Balanced Chains. In this section, we characterize a subclass of V* chains, namely the 
class of strongly aperiodic balanced chains. We first show that this class includes many of the chains 
that have been studied in the existing literature. Then, we prove that any aperiodic balanced 
chain belongs to the class V* . We also show that a balanced independent random chain is strongly 
aperiodic, thus concluding Theorem 1. 

Before continuing our analysis on balanced chains, let us discuss some of the well-known sub- 
classes of such chains: 

1. Balanced Bidirectional Chains: We say that an independent chain {M^(A;)} is a balanced 
bidirectional chain if there exists some a > such that E[Wjj(A;)] > aE[VFjj(/c)] for all /c > 1 
and i,j G [m]. These chains are in fact balanced, since for any S C [m] we have: 



E[Wss{k)] 



i£S,j€S 



> E 



aE[Wss{k)] . 



Examples of such chains are bounded bidirectional deterministic chains, which are the chains 
such that Aij{k) > implies Aji{k) > for all i.j G [m] and all k > 1, and the positive 
entries are uniformly bounded from below by some 7 > (i.e., Aij{k) > implies Aij{k) > 7 
for all i,j G [m] and all k > 1). In this case, for Aij(k) > 0, we have Aij{k) > 7 > 'yAji{k) 
and for Aij{k) = 0, we have Aji{k) = and, hence, in either of the cases Aij{k) > jAji{k). 
Therefore, bounded bidirectional chains are examples of balanced bidirectional chains. Such 
chains have been considered in [6, 1, 2]. 
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2. Chains with Common Steady State vr > 0: This ensemble consists of independent random 
chains {W{k)} such that E[7r-^14/'(A;)] = E[7r-^(A;)] for some stochastic vector vr > and all 
k > 1, which are generalizations of doubly stochastic chains, where we have vr = (e is a 
vector of ones). Doubly stochastic chains and the chains with a common steady state tt > 
have been studied in [7, 9, 8]. 

To show that a chain with a common steady state vr > is a balanced chain, let us prove the 
following lemma. 

Lemma 4. Let A be a stochastic matrix and ir > be a stochastic left- eigenvector of A corre- 
sponding to the unit eigenvalue, i.e., ir'^A = tt'^ . Then, A^g > A§g for any non-trivial S C [m], 
where VTmax = maxjg[^] VTj and VTmin = minjg[^] VTj. 

Proof. Let S C [m]. Since n'^A = vr^, we have 

(9) ^'^j = ^ T^iAij = ^ TTiAij + ^ TTiAij. 

On the other hand, since A is a stochastic matrix, we have vTj X^jgj^j Aij = tti. Therefore, 

(10) ^VTi = ^VTi ^ Ajj = ^ TTiAij + ^ TTiAij. 
i&S ieS je[m] i(^S,jGS ieS,j£S 

Comparing Eq. (9) and Eq. (10), we see that J2ieS jes'^i^ij ~ "^ies jes'^i^'i-j- Therefore, 

TTm'mAgg < TTiAij = TT^Aij < TTmax^SS- 

Hence, we have A^g > Agg for any non-trivial S C [m]. Q.E.D. 

The above lemma shows that a chain with a common steady state vr > is balanced with 
balancedness coefficient a = . In fact, the lemma yields a much more general result, as provided 
below. 

Theorem 5. Let {W{k)} be an independent random chain with a sequence {Tr{k)} of stochastic 
left-eigenvectors for the expected chain corresponding to the unit eigenvalue, i.e., ir'^ {k)E[W (k)] = 
■k'^ {k) for all k>l. Lf {7r{k)} > p* for some scalar p* > 0, then {W{k)} is a balanced chain with 
a balancedness coefficient a = — , ^ , . 

Proof. Since tt'^ {k)E[W (k)] = 7r'^(/c) for all /c > 1, by Lemma 4 we have 

^[Wssik)] > E[Wss{k)] for any non-trivial S C [m] and all A: > 1, 

By {7r(A;)} > p* > 0, it follows that7rmin(^) > P* for all k > 1. Since Tr(k) is a stochastic vector, it 
further follows 7rmax(^) < 1 — {m — l)7rmin(fe) < 1 — (m — l)p*. Therefore, for all k > 1, 

for any non-trivial S C [m]. Thus, {VF(A;)} is balanced with a balancedness coefficient a = 
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Theorem 5 not only characterizes a class of balanced chains, but it also provides an alternative 
characterization of the balancedness for these chains. Thus, instead of verifying Definition 1 for 
every nontrivial subset S C [m], for balancedness of independent random chains, it suffices to find 
a sequence {7r(fe)} of stochastic (unit) left-eigenvectors of the expected chain {E[PF(fe)]} such that 
the entries of the sequence do not vanish as time goes to infinity. 

4.1. Absolute Probability Sequence for Balanced Chains. In this section, we show that any in- 
dependent random chain that is strongly aperiodic and balanced must be in the class V* . 

The road map to prove this result is as follows: we first show that this result holds for deterministic 
chains with uniformly bounded positive entries where by a uniformly bounded chain {^(A:)} we 
mean that there exists a scalar 7 > such that Aij(k) > 7 whenever Aij{k) > 0. Then, using 
this result and geometric properties of the set of strongly aperiodic balanced chains, we prove the 
statement for deterministic chains, which immediately implies the result for independent random 
chains. 

To prove the result for uniformly bounded deterministic chains, we employ the technique that is 
used to prove Proposition 4 in [6]. However, the argument given in [6] needs some extensions to fit 
in our more general assumption of balancedness. 

Let {A{k)} be a deterministic chain of stochastic matrices. Let Sj{k) be the set of indices 
corresponding to the positive entries in the jth. column of A{k : 0), i.e., 

Sj{k) = {ee [m] I Aej{k : 0) > 0} for ah j € [m] and ah k > 0. 

Also, let Hj{k) be the minimum value of these positive entries, i.e., 

f^jik) = mii^ A^j(/c : 0) > 0. 

Lemma 5. Let {A{k)} be a strongly aperiodic balanced chain such that the positive entries in 
each A{k) are uniformly bounded from below by a scalar 7 > 0. Then, Sj{k) C Sj{k + 1) and 
fJ-jik) > 7l'^j('^)l~^ for all j G [m] and k>0. 

Proof. Let j G [m] be arbitrary but fixed. By induction on k, we prove that Sj{k) C Sj{k+1) for 
all > as well as the desired relation for fJ.j{k). For A; = 0, we have ^(0 : 0) = / by the definition, 
so ^^(O) = {j}. Then, A{1 : 0) = ^(1) and by the strongly aperiodic assumption on the chain 
{A{k)} we have Ajj{l) > 7, implying {j} = Sj{0) C Sj{l). Furthermore, we have |5'j(0)| — 1 = 
and ^j(O) = 1 = 7*^. Hence, the claim is true for A; = 0. 

Now suppose that the claim is true for some k > 0, and consider k + 1. Then, for any i E Sj{k), 
we have: 

m 

Aij{k + l:0) = Y^ Aie{k + l)Aej{k : 0) > A^ik + l)Aij{k : 0) > jfij{k) > 0. 
e=i 

Thus, i G Sj{k + 1), implying Sj{k) C Sj{k + 1). 

To show the relation for fij{k + 1), we consider two cases: 
Case ^Sj.(fc)5^.(fc)(A; + 1) = 0.' In this case for any i G Sj{k), we have: 

(11) A,j{k + l:0)= Aie{k)Aej{k:0)>fij{k) Au{k + 1) = fij{k), 

where the inequality follows from i G Sj{k) and ^5^(fc)5^.(fc)(A: + 1) =0, and the definition of 
fJ.j{k). Furthermore, by the balancedness of A{k) and ^5^(fe)5^{fc)(^ + 1) = 0, it follows that = 
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^5,(fe)5,(fc)(^ + 1) > «^5,(fc)5,(fc)(^ + 1) > 0. Hence, As^^^)s,ik)ik + 1) = 0. Thus, for any i G Sj{k), 
we have 

m 

Aj{k + 1:0) = Y^A,e{k + l)Aej{k : 0) = + l)Aej{k : 0) = 0, 

where the second equahty follows from Aij{k : 0) = for all i € Sj{k). Therefore, in this case we 
have Sj{k + 1) = Sj{k), which by (11) implies iij{k + 1) > fJ^jik). In view of Sj{k + 1) = Sj{k) and 
the inductive hypothesis, we further obtain 

//,-(A:) > 71^^(^)1-1 =71^^(^+1)1-1, 

implying fj.j{k + 1) > 'y\^3(^+^)\-^ . 

Case + 1) > 0; Since the chain is balanced, we have 

^Sj{k)S,{k)ik + 1) > aAs,(fc)s^.(fc)(/c + 1) > 0, 

implying that ^Sj(fc)5j{A:)(^) > 0- Therefore, by the uniform boundedness of {A{k)}, there exists 
^ € Sj{k) and i S Sj{k) such that A^^{k + 1) > 7. Hence, we have 

A^.{k + 1 : 0) > A^^ik + l)A^^{k : 0) > ^fij{k) = 7l^^('=)l, 

where the equality follows by the induction hypothesis. Thus, ^ S Sj{k + 1) while ^ ^ Sj{k), 
which implies \Sj{k + 1)| > |S'j(A;)| + 1. This, together with A^.{k + 1 : 0) > 7l^j('')l, yields 
fij{k + 1) > 7l-5^('^)l > j\Sj(k+i)\-i_ Q.E.D. 

The bound on Hj{k) of Lemma 5 implies that the bound for the nonnegative entries given in 
Proposition 4 of [6] can be reduced from ^'"^-'"+2 

Note that Lemma 5 holds for products A{k : tg) starting with any to > 0, (with appropriately 
defined Sj{k) and fij{k)). An immediate corollary of Lemma 5 is the following result. 

Corollary 3. Under the assumptions of Lemma 5, we have for all k > to > 0, 

-e^A{k : to) > min(-,7"^-i)e^, 
m m 

where e is the vector of ones and the inequality is to be understood entry-wise. 

Proof. Without loss of generality, let us assume that tg = 0. Then, by Lemma 5 we have 
—e^A^(k : 0) > —\Sj{k)\^^^^^^^^~^ for any j G [m], where A^ denotes the jth column of A. For 
7 € [0, 1], the function t 1— t- ^7*"! defined on [l,m] attains its minimum at either t = 1 or t = m. 
Therefore, ^e^A{k : 1) > min(^, 7™-i)e'^. Q.E.D. 

Now, we relax the assumption on the bounded entries in Corollary 3. 

Theorem 6. Let {A{k)} be a balanced and strongly aperiodic chain. Then, there is a scalar 
7 G (0, 1] such that A{k : 0) > min(^, 7'"-i)e'^ for all k>l. 
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Proof. Let a > be a balancedness coefficient for {A(k)} and let Aii(k) > /3 > for all i € [m] 
and k > 1. Further, let B^^^ be the set of balanced matrices with the balancedness coefficient a 
and strongly aperiodic matrices with a coefficient /3 > 0, i.e., 

(12) B„,^ := {Q G M™^™ | Q > 0, Qe = e, 

Qss — ^Qss non-trivial S C [?7t.],(5m > /3 for all i G [m]} . 

The description in relation (12) shows that B^^^ is a bounded polyhedral set in M*"^™. Let {Q^^^ G 
Ba,/3 I ^ G [n^^^]} be the set of extreme points of this polyhedral set indexed by the positive integers 
between 1 and n^^^, which is the total number of extreme points of Bq^^. 

Since A{k) G B^^^ for all A: > 1, we can write A{k) as a convex combination of the extreme 
points in Bq,^^, i.e., there exist coefficients X^{k) G [0, 1] such that 

(13) A{k) = \(A:)Q(^) with ^ A5(A:) = 1. 

5=1 5=1 

Now, consider the following independent random matrix process defined by: 

W{k) = Q(^) with probability X^{k) for all k > 1. 

In view of this definition any sample path of {VF(/c)} consists of extreme points of Bq,^^. Thus, every 
sample path of {VF(A:)} has a coefficient bounded by the minimum positive entry of the matrices in 
{Q^^^ G B^^^ I ^ G [^0^/3]}, denoted by 7 = j{a, (3) > 0, where 7 > since na,i3 is finite. Therefore, 
by Corollary 3, we have —e^W{k : to) > min(— , 7"^~^)e-^ for all k > to > 0. Furthermore, by 
Eq. (13) we have E[W(fc)] = A{k) for all fc > 1, implying 

—e^A{k : to) = —e^E[W{k : to)] > min ( — ,r 
m m \m 

which follows from {14^(A;)} being independent. Q.E.D. 

Based on the above results, we are ready to prove the main result for deterministic chains. 

Theorem 7. Any balanced and strongly aperiodic chain {^(A;)} is in the class V* . 

Proof. As pointed out in [4] for any chain {yl(/c)}, there exists a sequence {tr} of time indices, 
such that for all /c > 0, limr._>.oo A{tr '■ k) = Q{k) exists and, for any stochastic vector vr G M"^, the 
sequence {Q'^{k)TT} is an absolute probability sequence for {A{k)}. Since {A{k)} is a balanced and 
strongly aperiodic chain, by Theorem 6 it follows that 

LjQi^k) = — lim e^A{tr : k) > p*e^ for all /c > 0, 

with p* = min(^,7™-i) > 0. Thus, {^e'^Q{k)} is a uniformly bounded absolute probability 
sequence for {A{k)}. Q.E.D. 

The main result of this section follows immediately from Theorem 7. 

Theorem 8. Any balanced and strongly aperiodic independent random chain is in the class V* . 
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Proof. The proof follows immediately by noticing that, for an independent random chain, 
{W{k)}, any absolute probability sequence for the expected chain {E[VF(A;)]} is an absolute prob- 
ability process for {W{k)}. Q.E.D. 

As a result of Theorem 8 and Theorem 4, the proof of Theorem 1 follows immediately. In 
particular by Theorem 8, any independent random chain that is balanced and strongly aperiodic 
belongs to the class V* . Thus, the result follows by Theorem 4. 

5. Connection to Non-negative Matrix Theory. In this section, we show that Theorem 1 
is a generalization of the following well-known result in the non-negative matrix theory which plays 
a central role in the theory of ergodic Markov chains. 

Lemma 6. ([5], page 46) For an aperiodic and irreducible stochastic matrix A, limfc_>oo exists 
and it is equal to a rank one stochastic matrix. 



Recall that a stochastic matrix A is irreducible if there is no permutation matrix P such that 

P'^AP ■ 



X Y 
Z 



where X, Y, Z are i x i, i x [m — i), and {m — i) x (m — i) matrices for some i G [m — 1] and is 
the (m — i) X i matrix with all entries equal to zero. 

Let us reformulate irreducibility using the tools we have developed in this paper. 

Lemma 7. A stochastic matrix A is an irreducible matrix if and only if the static chain {A\ is 
balanced and its infinite flow graph is connected. 

Proof. By the definition, a matrix A is irreducible if there is no permutation matrix P such 
that 

" X Y 



P'^AP 



Z 



Since A is a non-negative matrix, we have that A is reducible if and only if there exists a subset 
S = {1, . . . , i} for some i € [m — 1], such that 

i}=[P^AP]ss= Yl edP^AP]ej= Yl = E 

where = {j G [m] | Pci = Cj} (which is a singleton since P is a permutation matrix) and 
R = Wi I i € S}. Thus, A is irreducible if and only if Agg > for all non-trivial S C [m]. 
Therefore, by letting 

AaQ 

a = mm — — , 

scH Ass 

and noting that a > 0, we conclude that {A} is balanced with a balancedness coefficient a. Fur- 
thermore, since Ag^ -\- Agg > Agg > for all nontrivial S C [m], it follows that the infinite fiow 
graph of {A} is connected. 

Now, suppose that {A} is balanced and its infinite fiow graph of {A} is connected. Then, 
Agg > Oor^^^ > for all non-trivial S C [m]. By the balancedness of the chain it follows 
that mm{Agg, Agg) > for any non-trivial S C [m], implying that A is irreducible. Q.E.D. 
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Note that for an aperiodic A, we can always find some h > 1 such that A'^^ > 7 > for all 
i € [m]. Thus, based on Theorem 1, we have the following extension of Lemma 6 for independent 
random chains. 

Theorem 9. Let {W{k)} be a balanced and strongly aperiodic independent random chain with 
a connected infinite flow graph. Then, for any to > 0, the product W{k : to) converges to a rank 
one stochastic matrix almost surely (as k goes to infinity). Moreover, if {W{k)} does not have the 
infinite flow property, the product W{k : to) almost surely converges to a (random) matrix that has 
rank at most r for any t^ > 0, where r is the number of connected components of the infinite flow 
graph of {^[W{k)\]. 

Proof. The result follows immediately from Theorem 1. Q.E.D. 

An immediate consequence of Theorem 9 is a generalization of Lemma 6 to inhomogeneous 
chains. 

Corollary 4. Let {A{k)} be a balanced and strongly aperiodic stochastic chain. Then, A[oo : 
to) = limfc^oo^(^ '■ to) exists for all to > 0. Moreover, A[oo : to) is a rank one matrix for all to > 
if and only if the infinite flow graph of {A{k)} is connected. 
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